[Previous] [Next] [Index]
[Thread]
Re: Site Scaning & IP graps
Content-Transfer-Encoding=8bit
X-Date Fri Mar 22 11:32:05 GMT 1996
X-Mailer: ELM [version 2.4 PL22]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Content-Length: 943
In a previous mail , KGANNON@dit.ie wrote :
>
> Can anyone tell me what the best way of detecting if a spider has had a look
> at your server i.e is there a list of common spiders.
>
> The story is there is a server here on site and while looking at the logs for
> my own page I saw a number of sites that I have never heard off. The thing
> is that there is no DNS entry for the server and it has not been advertised
> only 10 or so people on site even know it exists .
>
> I am just a little curious as to how people know about the server .
>
If you are interested by robots, you may want to check the following
document:
http://info.webcrawler.com/mak/projects/robots/active.htm
This is a list of *known* robots.
www security faq's :
http://www-genome.wi.mit.edu/WWW/faqs/www-security-faq.htm
--
Philippe Parmentier E-mail : P.Parmentier@ecmwf.int
Snail : ECMWF, Shinfield Park, Reading, Berkshire RG2 9AX, U.K.
References: